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STEPUP PROCEDURES CONTROLLING GENERALIZED FWER 
AND GENERALIZED FDR 1 

By San at K. Sarkar 

Temple University 

In many applications of multiple hypothesis testing where more 
than one false rejection can be tolerated, procedures controlling error 
rates measuring at least k false rejections, instead of at least one, for 
some fixed k > 1 can potentially increase the ability of a procedure 
to detect false null hypotheses. The fc-FWER, a generalized version 
of the usual familywise error rate (FWER), is such an error rate 
that has recently been introduced in the literature and procedures 
controlling it have been proposed. A further generalization of a result 
on the fc-FWER is provided in this article. In addition, an alternative 
and less conservative notion of error rate, the fc-FDR, is introduced 
in the same spirit as the fc-FWER by generalizing the usual false 
discovery rate (FDR). A fc-FWER procedure is constructed given 
any set of increasing constants by utilizing the fcth order joint null 
distributions of the p-values without assuming any specific form of 
dependence among all the p- values. Procedures controlling the fc-FDR 
are also developed by using the fcth order joint null distributions of 
the p- values, first assuming that the sets of null and nonnull p- values 
are mutually independent or they are jointly positively dependent in 
the sense of being multivariate totally positive of order two (MTP2) 
and then discarding that assumption about the overall dependence 
among the p- values. 

1. Introduction. Having realized that the traditional idea of controlling 
the familywise error rate (FWER), which is the probability of rejecting at 
least one true null hypothesis, is too stringent to use when a large num- 
ber of hypotheses are simultaneously tested, researchers have focused in the 
last decade on defining alternative less stringent error rates and developing 
methods that control them. The false discovery rate (FDR), which is the 
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expected proportion of falsely rejected null hypotheses and which was intro- 
duced by Benjamini and Hochberg [1], is the first of these that has received 
considerable attention [2, 3, 4, 5, 6, 12, 15, 16, 17, 21, 22]. Recently, the 
ideas of controlling the probabilities of falsely rejecting at least k null hy- 
potheses, which is the fc-FWER, and the false discovery proportion (FDP) 
exceeding a certain threshold 7 € [0, 1) have been introduced as alterna- 
tives to the FWER and methods controlling these new error rates have been 
suggested [10, 11, 13, 18, 23]. 

Sarkar [18] developed single-step and stepwise /c-FWER procedures utiliz- 
ing the kth order joint null distributions of the test statistics. He generalized 
the Bonferroni single-step procedure and obtained its Holm [8] type improve- 
ment, thereby providing a generalized fc-FWER stepdown procedure. These 
are different from the corresponding procedures of Lehmann and Romano 
[11]. He then generalized the procedure of Hochberg [7] by using a stepup 
procedure with the same critical values as those of his generalized Holm 
procedure. He proved that this generalized version of Hochberg's procedure 
controls the fc-FWER by making use of a generalized Simes' inequality he 
obtained for statistics that are positively dependent in the sense of being 
multivariate totally positive of order two (MTP2), a condition due to Karlin 
and Rinott [9] and often shared by test statistics in multiple testing. These 
alternative procedures are often more powerful than those in [11] based on 
marginal distributions, especially when the statistics are close to being in- 
dependent. 

In this article we continue our research in the line of [18] and develop newer 
procedures using kth. order joint null distributions of the test statistics. First, 
we generalize the method of Romano and Shaikh [13] and construct a stepup 
fc-FWER procedure given any set of increasing constants. Second, which 
is more interesting, we introduce a less conservative notion of error rate 
than the fc-FWER, which is the fc-FDR (to be defined later in this section), 
by generalizing the usual FDR in the same spirit as the fc-FWER, and 
provide newer stepup procedures that control it. The fc-FWER procedure 
is constructed without assuming any specific overall dependence among the 
test statistics. The fc-FDR procedures are derived under two scenarios-first, 
under the assumption that either the test statistics corresponding to the 
true null hypotheses are independent of those corresponding to the false null 
hypotheses or they are jointly MTP2, and then without that assumption. 

Let us denote by V and R the total number of false rejections and the 
total number of rejections, respectively, of null hypotheses. Then 



We will generalize it further in this article in terms of the following measure: 



(1.1) 



£>FWER = Pr{V > k}. 



(1.2) /c-FDR = E(k-FDP), where /c-FDP = I ^ 




otherwise. 
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The concept of /c-FDR has not been considered before, as far as we know, 
even though consideration of it, instead of the /c-FWER, appears to be a 
more natural extension of the idea of using the FDR as a less restrictive error 
rate than the FWER. Clearly, it reduces to the usual FDR when k = 1, and, 
more importantly, as /c-FDR < /c-FWER, controlling it would be a less 
conservative approach than controlling the /c-FWER. 

The construction of the fc-FWER stepup procedure is provided in Sec- 
tion 2. Section 3 is devoted to the derivation of the /c-FDR stepup proce- 
dures. The /c-FDR procedures are generalized versions of the usual FDR 
procedures of Benjamini and Hochberg [1] and Benjamini and Yekutieli [2], 
referred to as the generalized BH and generalized BY procedures, respec- 
tively, in this paper. The generalized BH version of the /c-FDR procedure 
provides uniformly better control of the /c-FDR, as it is intended to do so, 
than the generalized Hochberg procedure in [18] that, being a /c-FWER pro- 
cedure, controls the /c-FDR under the same distributional setup. As /c-FDR 
< FDR, an FDR procedure, such as the original BH procedure, also controls 
the /c-FDR. However, as our simulation studies indicate, although the gen- 
eralized BH /c-FDR procedure does not seem to outperform the original BH 
procedure when the number of true null hypotheses is small, its performance 
is much better when this number is relatively large and the test statistics 
are not highly dependent on each other. 

Sarkar [18] generalized Simes' test [20] by controlling the probability of 
at least k, instead of one, false rejections under the intersection of the null 
hypotheses. Interestingly, unlike the usual BH procedure, its generalized 
version that controls the /c-FDR is not based on these generalized Simes' 
critical values. In fact, we prove that the stepup procedure with the gener- 
alized Simes' critical values does not control the /c-FDR. 

Before we proceed to develop procedures with a control of the /c-FWER 
or /c-FDR, we recall here the definitions of stepdown and stepup procedures. 
Consider testing n null hypotheses H\, . . . ,H n simultaneously against cer- 
tain alternatives using their p- values P±, . . . ,P n , respectively. Let P\ :n < ■ ■ ■ < 
P n:n denote the ordered p- values. Then, given some critical values a\ < ■ ■ ■ < 
a n , a stepdown procedure accepts Hi for all i > jsD and rejects the rest, 
where jsD = nuni<j< n {i : Pj :n > oti}, if the minimum exists; otherwise, it re- 
jects all the hypotheses. A stepup procedure, on the other hand, rejects Hi 
for all i < jsu and accepts the rest, where jsu = maxi<j< n {i : Pj :n < Oj}, if 
the maximum exists; otherwise, it will accept all the hypotheses. These can 
be generalized by considering a\ = ■ ■ ■ = for some fixed 1 < k < n. 

2. fc-FWER controlling stepup procedure. In this section we consider 
developing a stepup procedure with a control of the /c-FWER at a start- 
ing with any increasing set of constants and using the /cth order joint null 
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distribution of the p- values. This is an attempt to generalize the idea of Ro- 
mano and Shaikh [13] that uses only the marginal p-values. First, we recall 
the following inequality from [18] that holds for any set of random variables 
Xx, . . . , X n , not necessarily p- values. 



Lemma 2.1. Let C k = {J: J C {1, . . . ,n}, \J\ = k} and Oj 



k,. . . ,n. Then, given any set of constants c k < • • • < c n , and 1 < k < n, we 
have 



(2.1) 



Prl \J (Xr. n <Ci)\<J2 PrivaaxXj < c k 
li=k J Jec k 1 j 

n - f 

+ ^ G ? rl X] Pr j Ci ~ l < maxX j - C > 
i=k+l jGC fc ^ 



Remark 2.1. The above lemma generalizes Lemma 3.1 of Lehmann 
and Romano [11]. When the fcth-order joint distributions are identical with 
Gk{x) = Pr{maxj g jXj < x} for all J G then it reduces to 



Prl {J(X itn <Ci)\< 



<i=k 



G k (c k )+ ar 1 {G k {c l )-G k {c i ^ 1 )} 



i=k+l 



(2.2) 

Considering k = 1 and uniform p-values, one gets the inequality given in 
[11]. 

We are now ready to describe our method of constructing a fc-FWER 
stepup procedure given any set of increasing constants a[ < ■ ■ ■ < a' n . We 
will, however, assume that the A:th-order joint null distributions of the p- 
values are identical. Let F k (x) = Pr{maxj e j Pj < x} for all J £C k and 

Dl 



(2.3) 
where 

(2.4) 



k,n 



max S' k (n ), 

K<no<n 



S Ln( n o) 



no 
k 



FkK 



n—no+k) 



+ a i X { F k{^'n-n +i) ~ F k{at n - no +i-l)} 

i=k+l 

and the probabilities are determined under the null hypotheses. 

Theorem 2.1. Given any set of constants at < ■ • ■ < a' n and < a < 1, 
consider the stepup procedure with the critical values a\ < • • • < a n satisfying 
Fk{oii) = aF k ( 

a ivk)/ F)'k n> * — 1, . . . ,n, where i V k — max(i, k) . This controls 
the k-FWER at a. 
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Proof. Assume without any loss of generality that the first no p- values 



correspond to the true null hypotheses. Let Pi 



"0 



< 



<p, 



n :n 



be the or- 



dered versions of these p-values. Assume that uq > k; otherwise, the k- 
FWER is zero and, hence, is trivially controlled. From [13] and using in- 
equality (2.2), we see that the fc-FWER of the stepup procedure in the 
theorem satisfies 



fc-FWER < Pr 




i:no 




(2.5) < 



( n 

u 



no 



J2 a i 1 {Fk{a n -n +i) ~ F k (a n - no+i ^i)} 



i=k+l 



ot- 



S 'k,n( n 0) 



which proves the theorem. □ 



Remark 2.2. Sarkar [18] proposed a stepdown procedure with the crit- 
ical values a± < ■ ■ ■ < a n satisfying Fk{a{) = a/a n+ k-i\/k, i = 1, ■ ■ . ,n, and 
proved that it controls the fc-FWER without assuming any overall depen- 
dence structure among the test statistics. It is alternative to, and often 
more powerful than, the stepdown procedure with the critical values = 
ka/(n + k — j V k), i = 1, . . . , n, that Lehmann and Romano [11] proposed. 
Sarkar [18] also showed that a stepup procedure with the same critical values 
as those of his A;-FWER stepdown procedure can also control the /c-FWER 
when the test statistics are all jointly MTP2. This generalizes Hochberg's 
procedure and its FWER control property under similar positive dependence 
condition; see [7, 14, 19]. When such MTP2 condition does not hold for the 
overall joint distribution of the test statistics, Theorem 2.1 says that one can 
appropriately rescale these critical values before using the stepup procedure 
in order to control the /c-FWER. 



3. fc-FDR controlling stepup procedures. In this section we will con- 
struct stepup procedures that control the fc-FDR using the feth-order joint 
null distribution of the p- values under two different scenarios, first when the 
sets of null and non-null p- values are known to be either mutually indepen- 
dent or are jointly dependent in the sense being MTP2 and second when no 
such condition is known. 



3.1. Generalized BH procedure. Toward developing the generalized BH 
version of our /c-FDR procedure, we first obtain a result providing a con- 
venient expression for the upper bound of the /c-FDR of a general stepup 
procedure. To that end, we will be using the following notation. 
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For a stepup procedure with the critical values a\ < • ■ • < a n , let R^-q' '"' 

and Vn-q' ' denote respectively the number of rejections and the num- 
ber of false rejections of null hypotheses when the stepup procedure based 
on the subset of p- values {Pi , . . . , P re } \ {P^ , . . . , Pj } and the critical values 
a q+ \ < ■ ■ ■ < u n is used. Let {Hi,i 6 Iq} be the set of true null hypotheses 
(|Io| = no). It is assumed that no > k; otherwise, the /c-FDR = 0, and hence, 
there is nothing to prove. 

Moreover, we will be using the following inequality. 

Lemma 3.1. In testing n null hypotheses, of which n$>k are true, let 
V n and R n denote the number of false rejections and the total number of 
rejections, respectively. Then, we have 

(3.1) I(Rn = r,V n >k)< {n ~ r+ 7 k)Vn I(R n = r,V n > k), 

for all 1 <k <r <n. 

Proof. The lemma follows by combining the following two inequalities: 
(i) V n (no — V Tl + k)> nok, which holds when k <V n < no, and (ii) no — V n < 
n — R n , which is true as R n — V n < n — no • □ 

Since V = J2iei I{H rejected), we have 
/c-FDP 

V n 1 

= ^/(^>A:) = ^-^/(P<a r , R = r, V>k) 

r=k i&h) 



n 1 



ElE^^' i£$=r-l, Vt^k-l) 



r 

r=k i£lo 



< X] {n — r + k—1) 



=k r(k-l)(n -l) 

x J2 V^Z?I{Pi < Or, R ( ~_!l = r - 1, F re ( Z? >k — l) 
iei 

(n — r + k — 1) 



it 



^r(fc-l)(no-l) 
(3.2) x ^(max(P, P,) < a r , R { ~X j) = r - 2, V^' j) >k-2) 

< 
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E{n — r + k — l)(n — r + k — 2) • • • (n — r + 1) 
r(k - i)(fc - 2) • ■ • l(n - 1) • • • (n - fc + 1) 



r=fc 

x ^ /(max(P il ,...,P i J<a r ,i?^'-'- ife) =r-fc, 

Vt-k-~ ik) >0) 

kn a n+k - r sr^ T ( D ^ D (-J) ,\ 
= > — : r > i max Fi < a r , K„ . . = r — fc , 

where C° = { J : J C I , | J\ = k} and R^-k 1S the number of rejections for the 
stepup procedure based on {Pi, i £ J c } and the critical values a^+i < • • < 
OO- 
LEMMA 3.2. For a stepup procedure with critical values a± < • • • < a n , 
we have 



/c-FDR < V PrimaxP,- < a k \ 

na n I *6J 



""o r=fc+1JeC o 



(3.3) 



M^i-fc >r-fc |Pi,ie J} 

f a n+k _ r I (mayL i( ; j Pi < g r ) 
\ r{n + k — r) 

(r — l)(n + k — r + 1) 

Proof. The inequality (3.2) yields 
fc-FDP 

kriQ a n+k _ r T ( D / dC-- 7 ) ^ ; 

< > — ; ; r > I max Pi < a r , R„ ,>r-k 

~ a no ^ k r(n + k-r) \ieJ 



> ; r > I ( maxPj < a r , iti > r - k + 1 

a nn ^ r(n + k-r) ^„ V *eJ n fc 

k 

(3.4) = ^£l(nuKJ*<a* 
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_l_ fere y y 



x 



a n+k ^ r I (max ie j Pj < ay) 
r{n + k — r) 

_ Qn+fc~r+l-^( max igJ-P» < a r-l) 

(r - l)(n + k - r+ 1) 
from which we get the lemma by taking expectations on both sides. □ 

We now construct a stepup procedure that controls the /c-FDR assuming 
that the p- values have identical feth-order joint null distributions and that 
the sets of p-values corresponding to the true and false null hypotheses 
are either independent or all the p- values are jointly MTP2. The MTP2 
property is a positive dependence property that holds for many multivariate 
distributions arising in multiple testing, for example, multivariate normal 
with a common nonnegative correlation, certain mixtures of independent 
random variables, and so on; see, for example, [14, 15, 16, 19]. For the 
definition of MTP2 and some of the related results to be used to prove the 
next result, one can see [9, 18]. 

Theorem 3.1. Consider the stepup procedure with the critical values 
c*i < ■ ■ ■ <a n satisfying 

a 
a,, 



(3.5) F k { ai ) = I ^ n + k _^ a 



fori = l,...,k, 
for i = k, . . . ,n. 



The k-FDR of this procedure is controlled at a if either the null p-values are 
independent of the nonnull p-values or the p-values are jointly MTP2. 

Proof. Assuming first that the null and nonnull p-values are mutually 
independent, we have from Lemma 3.2 that 

k-FDR <^ Fk {a k ) 
n 

+ £ £ M ufc?>r-fc} 

"»o r=k+l JgC o 

(3.6) 

a n +k-rFk{o<-r) a n +k-r+iF k (a r -i) 



r(n+k — r) (r — l)(n + k — r + 1) 

which is less than or equal to noa/n, and hence controlled at a, if F k (a r ) is 
chosen as in (3.5) for r = 1, . . . , n. 
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When the null and nonnull p-values are not mutually independent, but 
are jointly MTP2, we will prove the theorem as follows. 

Let g(pi,i G J) be the density of Pi, i G J, for any fixed J G Then, each 
expectation in (3.3) can be expressed as 



E 



Pr{R { ~ J i >r-k\ Pi ,iE J} { a n +k -rI(^ £j P,<a r ) 

I r(n + k — r) 

a-a+k-r+i I(maxi 6 j Pi < a r -i) 



(r — l)(n + k — r + 1] 

(3.7) 

= e\s T)J {P u i G J)ij> r ,j(Pi,i G J)lfmaxPj < a, 

= F k {u r )E*{4> r ,j{Pi,ie J)^j(Pi,i£ J)}, 

where the last expectation is taken with respect to the density 

,„ n g(pi,i£ J) I (mayL i( z j pi <a r ) 
(3-8J ^77 T , 

and 

(j>r,j(pi,i G J) = M#i-fc > r - fc|i>< = Pi) i G J}, 

/„ n N , / ■ _ t\ «n+fc-r «n+fc-r+l^(max ie jpi < « r _l' 
(3.9 1pr,j{Pi,l£J) = —, —j v 7 TT7 —j -TT 

r(n + k — r) (r — l)(n + k — r + 1) 

Since 

n—k 
i=r— k 

where pi.jc < • • • < p n -k : j<= are the ordered values of the set {pi, i G J c } that 
are all increasing in each pi,i G J c , the indicator function /(JfCjj? >r — k) 
is a decreasing (coordinatewise) function of the pj's. Also, the function ^ rj j 
is an increasing function of the pj's. 

The marginal density g(pi,i G J) of {Pi,i G J} is MTP2. As I(maxj g jpj < 
a r ) is also MTP2, the density (3.8), being the product of two MTP2 func- 
tions, is MTP2. We will now invoke the following property of MTP2 random 
variables. Random variables that are MTP2 are positively associated; that is, 
any pair of functions of these random variables, both increasing or decreas- 
ing, are positively correlated. As I{R ~_f} >r — k) is a decreasing function, 
the conditional probability of this function given Pj = pi, i G J, which is (f> r ,j, 
is a decreasing function of each pi,i G J, because of the MTP2 property of 
the p-values. Therefore, we have 

E*{(f> r ,j{P h ieJ)i> rt j{Pi,ieJ)} 

(3.11) 

< E*{4> r j(Pi,i G J)}E {ip r j(Pi,i G J)}, 



(3.10) {R { n ^> r -k}= |J {pi:jc<a k+i }, 
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where 



(3.12) 



E*{lp r ,j(Pi,i€ J)} 



a n+k- 



Q-n+k—r+l 



F k (a r . 



r{n + k — r) (r — l)(n + k — r + 1) F k (a r ) 
yielding the following inequality, again from Lemma 3.2: 



(3.13) 



k-FBR<^F k (a k ) 
re 

kriQ 



+ — E E E *{4>r,j{Pi,i € J)} 



* n a r=k+lj£C o 



a n +k-rFk(a r ) a n +k-r+iFk(a r -i] 



r(n + k — r) (r — l)(n + k — r + 1) 

which is less than or equal to noa/n for F k (a r ) satisfying (3.5). This proves 
the theorem. □ 



When k = 1, Theorem 3.1 reduces to the result establishing the usual FDR 
controlling property of the BH procedure; see [2, 15]. Of course, the result 
for the BH procedure is slightly stronger in that its FDR is less than or equal 
to noa/n under positive regression dependence, a slightly weaker condition 
than the MTP2 condition, and is exactly noa/n under the independence 
case. 

More explicit expressions of the critical values in Theorem 3.1 are 
k(k - 1) • • • la 



(3.14) F k (oi) 



n(n — 1) • 
for i ■ 



■(n-fc + 1) 
1, . . . , k, 
i(k - l)(k 



2) •••la 



n(n — i + k — l)(n — i + k — 2) • • • (n — i + 1) 
for i = k, . . . , re. 



With k = 2, 



(3.15) 



2a 



n(n — 1) 
ia 



for 



for 



1,2, 



. re. 



re(re — i + 1) ' 

In our procedure, the critical values corresponding to the smallest k 



1 p- 



values could be chosen arbitrarily. In other words, we can always reject the 
null hypotheses corresponding to the smallest k — 1 p- values and still control 
the fc-FDR. Nevertheless, we have considered rejecting these null hypotheses 
based on certain critical values. By choosing these critical values all equal 
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to a*;, we not only preserve the monotonicity of the critical values, but also 
have the least conservative choice among such critical values. 

It is important to see that our procedure provides uniformly better control 
of the fc-FDR, as it is intended to do so, than the generalized Hochberg 
procedure in [18] (see Remark 2.2) that, being a /c-FWER procedure, also 
controls the /c-FDR. This is because 

i(n + k-i)a a 
( 3 - 16 ) -ZTZ ^ 



7 — ; 

for alH = k, . . . , n. The extent of this improvement can be seen in Figures 1 
and 2 that are based on a numerical study to be discussed later in this 
section. 

As mentioned in the Introduction, the original BH FDR procedure also 
controls the /c-FDR. To see how it compares with our procedure, let us 
consider independent p- values and k = 2. The ith critical value of our pro- 
cedure, [ia/n(n — i + l)] 1 / 2 , exceeds the corresponding critical value ia/n 
of the BH procedure as long as n/i(n — i + 1) > a, which holds for each i if 
4n/(n + l) 2 > a. For instance, when a = 0.05 and n does not exceed 80, or 
when q = 0.01 and n does not exceed 400, our procedure has larger critical 
values, and hence, provides uniformly better control of the 2-FDR, than the 
BH procedure. Thus, in this case, our procedure often performs better than 
the BH procedure. Figures 1 and 2 provide some insight into this comparison 
in other cases. 

Sarkar [18] has generalized Simes' test for testing the overall hypothesis 
Hq : nr=i Hi- He proposed rejecting Hq if Pj :n < a, V fc for at least one i = 
1, . . . , n, for a fixed 1 < k < n, where 

/ Q17 x rp r \ a i i{i-l)---(i-k + l) . 

(3.17) ^te) = -°= n(n _ 1) .., (n _ fc + 1) °. . = *,...,", 

and proved that it controls the probability of at least k false rejections 
under the intersection null hypothesis at a exactly under independence and 
conservatively if the p- values are MTP2. Interestingly, unlike what is known 
when k = 1, the generalized Simes' critical values do not always control the 
fc-FDR. We provide a proof of this in the following. 

Let n\=n — riQ be the number of false null hypotheses. Consider a situa- 
tion where these n\ hypotheses are false to the extent that the corresponding 
p- values are all < a±. In other words, consider a procedure that with proba- 
bility one rejects n\ null hypotheses before proceeding as a stepup procedure 
based on the uq p-values that are all known to correspond to the null hy- 
potheses and using the critical values a ni +i, . . . ,a n . Let Rq be the number 
of true null hypotheses that are rejected. Then the fc-FDR of this procedure 
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is 



fc-FDR = e{ -J^—I(R >k)}> ^-rP{R > k}, 

(J. 18) 

k { n ° 1 k 

Let us now consider k = 2 and assume that the null p- values are i.i.d. U (0, 1). 
Then, the right-hand side of (3.18) simplifies to 

(3.19) — - (1 - an 1+2 r° - n a ni+2 (l - a ni+2 ) n °- 1 ], 
ni + 2 

which may exceed a for the generalized Simes critical values in some in- 
stances. For example, with a = 0.05, no = 100 and n\ = 1, we have 

■(ni + 2)(ni + l)al 1 / 2 



Oini+2 



n{n — 1) 

from which we see that (3.19) is equal to 0.0692. 



0.00545, 



3.2. Generalized BY procedure. We now derive a fc-FDR procedure only 
under the assumption of identical fcth-order joint null distributions of the 
p- values, which generalizes the BY procedure in [2]. We proceed as in the 
construction of the generalized BH procedure, but modify those results to 
facilitate the development of a fc-FDR procedure under the present scenario. 
First, we have the following upper bound for the fc-FDP that follows from 
(3.2): 

(3.20) fc-FDP<^£l £ lU f P t <a r ,R^ = r-k 

n r =k J £ c° 

which yields the following modification of Lemma 3.2. 

Lemma 3.3. For a stepup procedure with critical values a± < • ■ • < a n , 
we have 



k-FBR < V Prlm&xPi < a k 

na n f^ 



u n 

~na,, ^ ^ 



(3.21) 



r=k+l j e c° 



M^Lfe >r-k\Pi,ieJ} 

I(m&x i( zjPi < a r ) 
r 

_ I(m&X ie jPi < a. r -i) V 
r-1 J 
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Assuming now that the A:-th order joint null distributions are identical, 
we have the following theorem. 

Theorem 3.2. The stepup procedure with the critical values a\ < ■ ■ ■ < 
a n satisfying 

(3.22) W = I?l^n ■ < = l,...,n, 

controls the k-FDR at a. 



Proof. Lemma 3.3 yields the inequality 

(3.23) k-FDR < r ^ Fk {a k ) + k -^ £ V fe (a r ) - ^K-i)], 
n n , , r 

r=k+l 

which is equal to n^a/n when the critical values satisfying (3.22) are used. 
□ 



When k = 1, Theorem 3.2 reduces to the known result establishing the 
FDR control of the BY procedure [2]. When k = 2, the critical values of the 
generalized BY procedure are given by 

(3.24) F 2 (a i ) = ( * V ffi i = l,...,n. 

n(n - 1) Er=2 1 A 

3.3. A numerical study. We conducted a numerical study to investigate 
the extent of improvement offered by the generalized BH version of our k- 
FDR procedure in controlling the /c-FDR over the generalized Hochberg and 
the original BH procedures when k = 2. 

We generated n = 100 dependent random variables Xj ~ iV(/ij,l), z = 
1, . . . , 100, with the same variance 1 and a common correlation p, which are 
known to be MTP2, and performed 100 hypothesis tests of fj, = against 
fj, = 2, using each of these three procedures with a = 0.05. The value of 2- 
FDP was then calculated for each procedure by setting no of the u^'s to 
zero and the remaining n\ of the Uj's to the value 2. The 2-FDR then was 
estimated by averaging the 2-FDP values over 5000 iterations. We did these 
calculations in two different scenarios, when the X^s are independent (p = 0) 
and when they are weakly dependent (p = 0.10). Figures 1 and 2 compare 
the simulated 2-FDRs of these procedures for different values no when p = 
and 0.10, respectively. 

Figures 1 and 2 indicate that while our procedure provides uniformly bet- 
ter control of the fc-FDR than the generalized Hochberg procedure, which 
we expected, the difference is, however, quite significant when no is neither 
very small nor very large. Compared to the original BH procedure, we notice 
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that our procedure works quite well when the test statistics are independent 
or close to being independent and no is relatively large, that is, when rela- 
tively few of the null hypotheses are expected to be false. With increasing 
dependence among the test statistics, our procedure loses its edge over the 
BH procedure for small no- 

4. Concluding remarks. Generalizing traditional error rates to make them 
more appropriate in situations where one is willing to tolerate more than one 
false rejection and is wishing to increase the ability of procedures to detect 
false null hypotheses using error rates that allow such rejections has become 
an increasingly important idea in multiple testing, because of its relevance 
in testing a large number of hypotheses, as in microarray studies. The work 
done in this article makes an important contribution in this area. In addi- 




FlG. 1. Comparison of 2-FDRs with p = 0.0 (Generalized BH: 
Hochberg: Original BH: — ). 



; Generalized 




tion to generalizing previous work on the fc-FWER, the notion of FDR has 
been generalized for the first time and procedures controlling it have been 
developed in this article. We believe we have opened the door in this article 
for further research in multiple testing, particularly toward developing the 
theory and methodology of false discovery rate. There are several interesting 
generalizations of results related to the original concept of FDR that could 
potentially be developed along the line of this article. 

Acknowledgments. The author thanks Zijiang Yang for doing the nu- 
merical calculations and a referee for giving helpful suggestions. 
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